Creedo—Scalable and Repeatable Extrinsic Evaluation for Pattern Discovery Systems by Online User Studies

We present Creedo—a conceptual framework along with an accompanying software implementation for empirically evaluating knowledge discovery systems. Creedo provides a language for the creation of study designs that test how well different test systems support real users to perform certain data analysis tasks. These designs are scalable and repeatable, i.e., after their creation, a study can be carried out any number of times and with an arbitrary high number of participants without consuming valuable resources such as the time of supervision personnel. Building on the conceptual framework, the accompanying web application, which is freely available at Bitbucket, supports data mining researchers in all central tasks for conducting a user study: in embedding their ideas into an operational data analysis environment, in assigning and monitoring tasks for study participants, and in evaluating the results. The implementation as web application enables large scale and geographically distributed studies, in which, nevertheless, all study participants essentially have an identical user-experience.